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We analyze how difficult it is to synchronize to a periodic sequence whose structure is known, 
when an observer is initially unaware of the sequence's phase. We examine the transient information 
T, a recently introduced information-theoretic quantity that measures the uncertainty an observer 
experiences while synchronizing to a sequence. We also consider the synchronization time r, which 
is the average number of measurements required to infer the phase of a periodic signal. We calculate 
T and r for all periodic sequences up to and including period 23. We show which sequences 
of a given period have the maximum and minimum possible T and r values, develop analytic 
expressions for the extreme values, and show that in these cases the transient information is the 
product of the total phase information and the synchronization time. Despite the latter result, our 
analyses demonstrate that the transient information and synchronization time capture different and 
complementary structural properties of individual periodic sequences — properties, moreover, that 
are distinct from source entropy rate and mutual information measures, such as the excess entropy. 

PACS: 02.50.Ey 02.50.Ga 05.45.-a 05.45. Tp 89.75.Kd; 
Santa Fe Institute Working Paper 02-08-043 



I. INTRODUCTION 



Imagine you are about to begin observing a sequence 
of events. You know the sequence is periodic; you even 
know the particular pattern that will repeat. However, 
you do not know what phase the sequence is in. How 
many observations, on average, would you have to make 
before you know with certainty the sequence's phase? 
And, as you obtain this certainty, how uncertain are you 
about the phase? Will the answers to these questions 
be the same for all sequences of a given period or are 
there differences between such sequences? What struc- 
tural properties of a sequence determine the difhculty of 
synchronization? Here, we answer these questions. 

A natural place to begin is with information theory, 
since it has long been used to analyze the statistical prop- 
erties of sequences that arise in a variety of settings, in- 
cluding dynamical systems, time-series analysis, statisti- 
cal mechanics, signal processing, and cryptography 
One of the central quantities in these analyses is the en- 
tropy rate /i^, the long-time measure of unpredictabil- 
ity of sequences produced by an information source. Al- 
though useful and important for quantifying randomness, 
/i^ does not capture a sequence's structural properties: 
its correlation, memory, or statistical complexity. For- 
tunately, today there are scores of measures of these lat- 
ter properties. For example, on the information theoretic 
side, an oft-used measure of memory is the excess entropy 
E — the time-averaged, "all-point" mutual information 



between a sequence's past and future 

For any sequence of period P, it is well known that 
the entropy rate vanishes and the excess entropy 
E = log2 P, since a periodic sequence is asymptoti- 
cally predictable and since log2 P bits of information are 
needed to store in which of the P possible phases a sym- 
bol in the sequence is. Thus, all periodic sequences of 
the same period have the same entropy rate and excess 
entropy, and so these quantities are unable to capture 
structural differences between distinct sequences of the 
same period. In particular, and E cannot help an- 
swer the questions posed above. 

We turn instead to a recently introduced mea- 
sure, the transient information T ||^,^, which captures 
information-theoretic differences between sequences of a 
given period. We will also make use of the synchroniza- 
tion time r, defined as the average number of observa- 
tions needed to synchronize to a periodic information 
source. We shall see that there are indeed significant 
differences in the synchronization properties of periodic 
sequences. Furthermore, while T and t are related for 
extreme cases — minimum and maximum T and r se- 
quences — there is a wide range of T and r values for 
periodic sequences of the same period. 

In the following section we review information sources, 
entropy rate, and excess entropy. In Sec. Ill we define 



T and t more carefully. After a brief discussion of com- 
binatorics and symmetry types of periodic sequences in 
Sec. [V, in Sec. M we discuss methods used to calculate 
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T and r. In Sec. VI we present the results of exh aus- 



tivel y cal culating T and r up to period 23. In Sees. |VII 
and VIU , we investigate these results, developing ana- 
lytic expressions for the minimum and maximum T and 
T values of a given period, exploring relationships and 
bounds between the two synchronization measures. Fi- 
nally, in Sec. IX we summarize and interpret the results, 
pointing out several applications to coordination in mul- 
tiagent systems. 



II. ENTROPY RATE AND EXCESS ENTROPY 

A. Information Sources 

We shall be concerned with a one-dimensional infinite 
sequence of variables: 



(1) 



Here, the S'f's are random variables that range over a 
finite set A of alphabet symbols. In general, A = 
{0, 1, . . . , A; — 1}, although in the following, we will re- 
strict ourselves to binary sequences. We denote a block 
or word of L consecutive variables by = Si . . . Sl- 
We follow the convention that a capital letter refers to a 
random variable, while a lowercase letter denotes a par- 
ticular value of that variable. Thus, = sqSi ■ ■ ■ sl-i, 
denotes a particular symbol block of length L. 

We assume that the underlying information source is 
described by a shift-invariant measure _n on infinite se- 
quences • ■ • s_2S-iSoSiS2 ■ ■ ■ ;st e A 0. The measure 
/i induces a family of distributions, {Pr(st+i, . . . , St+L) : 
St G A}, where Pr(st) denotes the probability that at 
time t the random variable St takes on the particular 
value St € A and Pr{st+i, ■ ■ ■ , St+L) denotes the joint 
probability over blocks of L consecutive symbols. We as- 
sume that the source is stationary; Pr(sf_|_i, . . . , St+h) ~ 
Pr(si, . . . ,sl). 

We are interested in periodic sequences; a sequence 
is periodic of period P if = Si+p for all i. The 
prime period of the sequence is the smallest such P. 
We can specify a periodic sequence by giving the small- 
est word — the prime word — that is exactly repeated. 
The prime word is aperiodic, necessarily. For example, 
. . . 1100110011001100 ... is period 8 (and period 16 and 
so on), but has prime word 1100 and prime period 4. 



B. Entropy Growth and Entropy Rate 

Here, we give a brief review of the information- 
theoretic description of sequences, concentrating on those 
that are periodic. For more detail about information- 
theoretic measures of uncertainty and structure in the 
context of general one-dimensional random sequences, 
see, e.g., Refs. ||Jl^,|ll|] and references therein. 



by: 



The total Shannon entropy of length-L blocks is defined 



H{L) 



Pr(s^)log2Pr(s^ 



(2) 



where L > 0. The sum is understood to run over all pos- 
sible blocks of L consecutive symbols. The units of H{L) 
are bits. The entropy H{L) measures the average uncer- 
tainty in identifying one sequence in the set of length-i 
sequences. Equivalently, H{L) tells us how many yes-no 
questions, on average, are needed to determine the value 
of S^ . Another way to state this is that H{L) sets a lower 
bound on the size of the code, measured in bits, needed to 
encode successive outcomes of the random variable 5'^. 
For more on this coding interpretation of the Shannon 
entropy, see, e.g., Ref. 

The general behavior of H{L) as a function of L for 
periodic information sources is shown schematically in 
Fig. 0. Note that we define iJ(0) = 0. 




FIG. 1. Total Shannon entropy growth: A schematic plot 
of H{L) versus L. The entropy H{L) of L-blocks increases 
monotonically and approaches the asymptote E-|-ftpL, where 
E is the excess entropy and /i^ is the source entropy rate. The 
shaded area is the transient information T. 

The entropy rate hf^ is the rate of increase with respect 
to L of the total Shannon entropy in the large-i limit: 



h„ 



lim 



H{L) 



(3) 



where fi denotes the measure over infinite sequences that 
induces the L-block joint distribution Pr(S'^); the units 
are bits per symbol. One can also define a finite-L ap- 
proximation to h^, 



h^{L) = H{L)-H{L-1) 
= H[Sl\Sl-iSl-2 



■ So] 



(4) 
(5) 



where is the entropy of the random variable 

X conditioned on the random variable Y: H[X\Y] = 
— J2x,y y) log2 Pr(x|y). One can then show [g] that 
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hf^ = lim h^{L) 

L — >oo 



(6) 



E= lim /[S'_L+i, S'_L+2, •■■ 5*0; 5'i5'2 ••■ S'l] , (9) 

L — >oo 



Note that Eq. (Q) shows that hf^(L) may be viewed as 
the slope of H{L). For more on this point of view, see 
Ref. i_ 

Eq. (p) gives us another interpretation of /i^. The en- 
tropy rate quantifies the irreducible randomness or un- 
predictability of the sequences; measures the random- 
ness that persists even after the statistics of longer and 
longer blocks are taken into account. Since all periodic 
sequences are, ultimately, predictable, the entropy rate 
for all periodic sequences is ft-^ = 0. For a period-P se- 
quence h^{P) = and hf,{L) > 0, for i = 1, 2, . . . , P-1. 
This is illustrated in Fig. g for three different period-8 
sequences; when the slope of H{L) vanishes, h^{L) = 0. 
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FIG. 2. Shannon entropy growth curves H{L) for three dif- 
ferent period-8 sequences: 00010111, 00010101, and 00000001. 



C. Excess Entropy: Apparent Memory 



when the limit exists j|J^ . Eq. (||) says that E measures 
the amount of historical information stored in the present 
that is communicated to the future. 

Our focus here is on periodic sequences, for which it is 
easy to show from one or another of the above definitions 
that 

E = logaP . (10) 

That is, the excess entropy is the total phase informa- 
tion stored in the periodic source. As noted above, the 
entropy of a random variable gives the average number 
of yes- no questions (i.e., bits), needed to determine the 
outcome of the variable. Since the observer starts out 
equally ignorant of the initial phase, it follows that E is 
a lower bound on the amount of information (in bits) an 
observer must extract in order to know in which phase 
the source is — that is, in order for it to be synchronized 
to the source. 

The entropy growth curves H{L) for three different 
period-8 sequences are shown in Fig. ^j. Note that all 
curves asymptote to the same line, 'Ei+h^L. Since ~ 
(periodic sequences) and E = logj P = logj 8 = 3, the 
H{L) curves asymptote to the horizontal line H{L) — 3 
bits, when L > 8. 

Despite the fact that these three period-8 sequences 
have the same and E, Fig. ^ shows that their entropy 
growth curves H{L) are not the same; for example, they 
reach the linear asymptote at different L's. Do all period- 
8 curves have different H{L) behaviors? And do these 
different H{L) behaviors matter, or do and E suffice 
to characterize periodic sequences? 



The entropy rate measures the randomness of se- 
quences. A complementary quantity is the excess entropy 
E, which measures the deviations of finite-L estimates of 
hf^ from its asymptotic value: 

oo 

E ^ ^[/^^(i)-M , (7) 

L=l 

The units of E arc hits. Using this definition, one can 
show that the excess entropy is the subextensive part of 
H{L) (see, e.g., ||,|llHll,|); that is, 

E = lim [H{L) - hf,L] . (8) 

L — >oo 

This establishes a geometric interpretation for E, as 
shown in Fig. |l| it is the y-intercept of the line to 
which H{L) asymptotes. That is, when E is finite, H{L) 
asymptotes to E -I- /i^L. 

Another way to understand excess entropy is through 
its expression as a mutual information: the mutual infor- 
mation between the past and future semi-infinite halves 
of the chain of random variables: 



III. MEASURES OF SYNCHRONIZATION 

A. Transient Information 

To address these questions, we consider the transient 
information T, which was introduced by us in Ref. |^ 
and is defined by summing the deviations of H[L) from 
its linear asymptote: 

oo 

T ^ ^[E + /vi-i/(L)] . (11) 

L=0 

Note that the units of T are hits x symbols and that 
T > E. The transient information is the shaded area in 
Fig. 0. We refer to T as transient, since, unlike and E, 
it is dominated by nonasymptotic quantities: how H{L) 
behaves before it reaches the linear scaling form E + Zi^L. 
By inspection, one sees that the three period-8 sequences 
in Fig. ^ all have different T values. 
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As noted above, for finite-E processes H{L) scales as 
E + h^L for large L When this scaling form is at- 
tained, we say that the observer is synchronized to the 
source. In other words, when 

T{L) = B + h^L- H{L) = , (12) 

we say the observer is synchronized at length-L se- 
quences. The quantity T(L) provides a measure of the 
departure from synchronization. Note that T(L) is non- 
negative. Looking at Eq. we see that the transient 
information T may be written as a sum of the T(L)'s: 

oo 

T=^T(L). (13) 

To give the transient information a more precise in- 
terpretation, let us consider in more detail the synchro- 
nization scenario we have in mind. An observer begins 
making measurements of a sequence, seeing one symbol 
at a time. The observer knows the periodic sequence it 
is about to start seeing, but it doesn't know the phase. 
That is, the observer knows the period and the prime 
word that will be repeated. The task for the observer 
is to make measurements and determine the sequence's 
phase. Exact prediction is possible from this point on- 
wards, though not before. How uncertain is the observer 
during this synchronization process? 

To answer this question, we must introduce some addi- 
tional notation. Before the observer is synchronized, its 
knowledge is characterized by a distribution over the P 
possible phases of the periodic sequence. Equivalently, 
the sequence's phases may be viewed as the periodic 
source's internal states. Let ip denote a particular phase, 
and let $ denote the set of all phases. Clearly, |$| = P. 
Next, let Pr((^|s^) denote the probability, as inferred by 
the observer, that the sequence is in phase given that 
it has just seen the particular sequence of symbols . 
The entropy of the distribution of ip measures the ob- 
server's average uncertainty of the phase <i>. Averaging 
this uncertainty over the possible length-L observations, 
we obtain the average state-uncertainty: 

H{L) = -^Pr(s^)^Pr(^|s^)log2Pr(^|s^) . (14) 

The quantity 'H{L) can be used as a criterion for synchro- 
nization. The observer is synchronized to the sequence 
when 7i(L) = — that is, when it is completely certain 
about the phase, or internal state, g $ of the source 
generating the sequence. And so, when the condition in 
Eq. (HI) is met, we see that H(L) = 0, and the uncer- 
tainty associated with the next observation is 0. 

However, while the observer is still unsynchronized 
'H{L) > 0. The average total uncertainty experienced 
by the observer during the synchronization process is the 
total synchronization information S: 



p 

S^E^W- (15) 

The synchronization information measures the total un- 
certainty 'H{L) experienced by an observer during syn- 
chronization. For the periodic sequences under consider- 
ation here, it turns out that 

S = T . (16) 

That is, the transient information T is equal to the total 
synchronization information S. 

Eq. ( p^ is a special case of a theorem recently proved 
by us in Ref. and discussed further in Ref. [p|. H ere, 
we briefly sketch the main argument behind Eq. (|16|) . At 
L = 0, no measurements have been made and the ob- 
server posits that each phase is equally likely. The state- 
uncertainty is thus log2 P = E. After L observations 
have been made, the observer has gained, on average, 
H{L) bits of information about the phase of the process. 
As a result, the average state- uncertainty Ti.{L) is now 
E — H(L). Plugging this last observation into Eq. ([T^), 
Eq. (|l6|) follows from the definition of T, Eq. (0). 

As a result of Eq. (p^, the transient information T 
provides a direct measure of how difficult or confusing 
it is to synchronize to a periodic sequence. If a periodic 
sequence has a large T, then on average an observer will 
be highly uncertain about its phase during the synchro- 
nization process. The transient information measures a 
structural property of a sequence — a property captured 
neither by the entropy rate nor by the excess entropy. If 
the observer is in a position where it must take immediate 
action, it does not have the option of waiting for full syn- 
chronization. In this circumstance, the synchronization 
information S provides an information-theoretic average- 
case measure of the error incurred by the observer during 
the synchronization process. 

B. Synchronization Time 

In addition to measuring the total uncertainty expe- 
rienced during synchronization, we can ask a related 
question: On average, how many measurements must be 
made before the observer is certain in which state the 
source is? We call the number r of measurements, av- 
eraged over the P possible starting phases, the synchro- 
nization time. 

For example, consider the periodic sequence (0001)°°. 
There are four possible phases in which one might be- 
gin to observe this sequence. If the first symbols parsed 
are 1 — — — 0, then the observer is synchronized after 
the first observation (a 1). If the observer initially sees 
— 1 — — 0, it is synchronized after two symbols. If 
the observer sees either — 0— 1— or — — — 1, it 
is synchronized after three measurements. Each of these 
initial measurement sequences is equally likely. Thus, on 
average, it will take 
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1+2+3+3 , , 

T = = 2.25 17 

4 

measurements before the observer is synchronized to 
(0001)°°. 

Below, we conduct a survey of the transient informa- 
tion T and synchronization time t of periodic sequences. 
Although there is an overall scaling between them, per- 
haps somewhat surprisingly, we shall see that T and r 
capture fundamentally different properties of a sequence. 
We shall also see that within different sequences of a given 
period there are wide variations of T and r. First, how- 
ever, we pause to consider some combinatorial properties 
and symmetry classes of periodic sequences. 



These aperiodic words are also related to the so-called 
Lyndon words. A Lyndon word is aperiodic and the 
lexicographically least among its rotations. An unla- 
beled Lyndon word is aperiodic and lexicographically least 
among its rotations and relabelings. Eq. (p^), then, 
also counts the number of binary unlabeled Lyndon 
words. For more discussion, see Rcf. nm and sequence 
#A000048 of Ref. (l|. Titsworth @has pointed out 
that Eq. ( p^ also gives the number of distinct finite-state 
machines needed to produce all periodic binary sequences 
of prime period P. 

V. METHODS 



IV. ENTROPIC SYMMETRY TYPES AND 
COMBINATORICS OF PERIODIC SEQUENCES 

How many periodic sequences of a given prime period 
are there? Aperiodic length-P words that are related 
by an application of a cyclic permutation group Cp give 
the same periodic sequence. Thus, for our purposes — 
considering infinitely repeated periodic sequences — two 
length-P words related via the group operation Cp are 
equivalent; they yield identical infinite sequences. For 
example, the aperiodic words 001, 010, and 100 all yield 
the same, infinite, period-3 sequence. 

However, we are interested in looking for distinct H{L) 
versus L behaviors, and this induces another set of equiv- 
alences on the period-P sequences. Because H[X] de- 
pends only on the distribution of X, and not on the 
values X assumes, the entropy of a random variable is 
unchanged under a one-to-one mapping of alphabet sym- 
bols 1^. For the binary cases we are interested in here, 
this means that the mapping 1 leaves H{L) un- 
changed; for our purposes, the sequence (011)°° is iden- 
tical to (100)°°. In terms of group action, swapping 1 
and is the action of the symmetric group 5*1. So, any 
two sequences that arc based on binary aperiodic length- 
P words related to each other by some combination of 
and Cp will have identical H{L) versus L behavior. 

The number L2 of aperiodic binary words inequivalent 
under 5*1 x Cp is given by: 

d\P,d odd 

The sum runs over all odd positive divisors d oi n, and 
fi{m) is the Mobius function: fi{m) = 0, if to is the prod- 
uct of nondistinct primes; +1, if m is the product of an 
even number of distinct primes; and —1, otherwise. This 
result was first obtained by Fine jlj] in 1958 and simpli- 
fied a few years later by Gilbert and Riordan ||l5| . These 
words are also related to a version of the famous necklace 
problem from combinatorics: the words inequivalent un- 
der 5*1 X Cp are exactly those 2-color necklace sequences 
with prime period P where interchanging bead color is 
allowed, but turning the necklace over is not. 



We calculated the transient information T and syn- 
chronization time r for all distinct periodic sequences 
(unlabeled Lyndon words) up to and including period 
P = 23. To enumerate all of these words of a given 
period, we used the efficient algorithms available at 
Ref. @. 




FIG. 3. Parse tree for the period-5 sequence with prime 
word 00011. Unvisited paths are not shown. The synchroniz- 
ing words are shown with bold paths from the start node. 



Prime Word T r 

000001 6.97905 3.33333 

000011 5.37744 2.83333 

000101 5.58496 3.33333 

000111 4.83659 2.66667 

001011 4.83659 2.66667 



TABLE I. The transient information T and synchroniza- 
tion time r of all five distinct period-6 sequences. 
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To calculate T and r for a given length-P word, we 
begin by determining the frequency of occurrence of all 
its subwords. This is done by parsing the word and its 
P — 1 cyclic permutations into a tree, whose paths from 
the root to any node are the subwords and each of whose 
nodes contain the number of subwords found that lead 
to it from the root. 

An example is shown in Fig. ^ Consider the sequence 
whose prime word is the length-5 word 00011. We start 
at the tree root, indicated by the double circle at the top, 
and follow the leaves labeled with the appropriate sym- 
bol. Each time we cross a node we increment the count 
there by one. We repeat this procedure for all 5 cyclic 
permutations of 00011. 

Once the parse tree is built, T can be easily and ex- 
actly calculated. First, the node counts are turned into 
probabilities by normalizing — dividing each count by 
the period P. Reading across the tree at level i gives 
the probability of subwords of length £. In the P — 5 
example considered here, Pr(OO) = 2/5, Pr(Ol) = 1/5, 
Pr(lO) = 1/5, and Pr(ll) = 1/5. From these probabil- 
ities, the block entropies H{L), and, in turn, T, follow 
directly by using Eq. ( ITTl ) and noting that E = logj P 
and = 0. 

Calculating t is only a bit more involved. After build- 
ing the parse tree, we reparse each cyclic permutation 
of the word. As we reparse and proceed down the tree, 
we monitor the node counts. For each cyclic permuta- 
tion of the prime word, we follow the corresponding path 
from the root. When we come to the first node that has a 
count of 1, we are synchronized. That is, for the cyclic 
permutation of the word, the level £ at which we first en- 
counter a node count of 1 is the synchronization time 
for that permutation. The average synchronization time 
T is then simply: 

1 

^=pE-- (19) 

For the example in Fig. ^, we have 

r = i(3 3 -f 2 -I- 2 -f 2) = 2.4 . (20) 
5 

As for T, the above method yields an exact value for r. 

For example, in Table | we show the results of calcu- 
lating the transient information T and the synchroniza- 
tion time T for all five distinct period-6 sequences. (The 
reader may find it helpful to verify these results.) 

VI. EMPIRICAL RESULTS 

We empirically investigated the behavior of T and r 
by first exhaustively enumerating all distinct periodic se- 
quences up to and including period P = 23 and then ex- 
actly calculating these two quantities for each using the 



methods of the previous section. The results are sum- 
marized in Figs. |, |, |, 0, and |. In Table || we show, 
for periods 2 through 23, the number of distinct periodic 
sequences, the number of distinct T and t values, and 
the mean, minimum, and maximum T and t values. 

Based on these results, one can make a host of obser- 
vations. First, note that T and t are not the same for 
all sequences of a given period. This can be seen in our 
period-6 results of Table |, as well as in Figs. ^ and 
|6|. Thus, within a given period, there are many different 
synchronization behaviors. As noted before, these differ- 
ences are not accounted for by the entropy rate and 
the excess entropy E, since E and /i^ are the same for all 
sequences with the same period. 

45 I 1 1 1 1 — I 

40 ■ ■ 

I- 35 ■ : ■ 

o 

^ 30 ■ Mi' 




Period 

FIG. 4. Scatter plot of the transient information T for all 
distinct periodic sequences as a function of their prime pe- 
riod P, up to P = 22. The dashed lines are the maximum 
and minimum T values computed analytically; see text. The 
solid line gives the estimated average T value at the given 
period. 
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Period 

FIG. 5. Scatter plot of the synchronization time r for all 
distinct periodic sequences as a function of their period P, up 
to P = 22. The dashed lines are the maximum and minimum 
T values computed analytically; see text. The solid line gives 
the estimated average r value at the given period. 
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5.13293 


4.00000 


11.4375 


17 


3855 


1091 


12.5298 


10.4930 


30.6969 


104 


5.25312 


4.11765 


12.2353 


18 


7280 


1878 


12.9133 


10.9359 


33.1614 


104 


5.33134 


4.22222 


12.7778 


19 


13797 


3205 


13.3158 


11.3449 


35.6640 


132 


5.43299 


4.31579 


13.7368 


20 


26214 


5015 


13.6772 


11.7168 


38.2027 


143 


5.50798 


4.40000 


14.4500 


21 


49929 


10355 


14.0405 


12.0774 


40.7759 


153 


5.59134 


4.47619 


15.2381 


22 


95325 


16031 


14.3815 


12.4083 


43.3818 


191 


5.66223 


4.54545 


15.8182 


23 


182361 


27322 


14.7179 


12.7191 


46.0192 


207 


5.73570 


4.60870 


16.7391 



TABLE II. The number of distinct periodic sequences, the number of distinct T and r values, and the average, minimum, 
and maximum T and r values as a function of period, up to P = 23. 
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FIG. 6. The number of distinct periodic sequences, distinct 
T values, and distinct r values as a function of period P. Note 
the logarithmic scale on the vertical axis. 

Second, there are words with the same period that have 
identical T or identical r values. This can be seen in 
Fig. ^, in which we have plotted the count data shown in 
Table The number of distinct T and t values clearly 
grows less quickly than the total number of distinct peri- 
odic sequences. Note also that the number of distinct T 
and T values grow at different rates. This provides direct 
evidence that the transient information and the synchro- 
nization time measure different qualities. For example, 



in Table |[ note that two sequences have the same r but 
different T's. The converse also occurs, but much less 
frequently. There are sequences with the same T but 
different r values, but this does not occur until P = 12. 
As can be seen from both Table || and Fig. H, r tends 
to be substantially more degenerate than T; r is some- 
how a coarser measure of a sequence's synchronization 
properties. 



Third, note that for a given period there is a consider- 
able range of transient information and synchronization 
time values. This can be seen in Figs. ^ and |^, where 
the minimum and maximum T and r values are shown 
as dashed lines. This suggests that there are significant 
differences in synchronization behaviors among different 
sequences with the same period. We will return to the 
question of minimum and maximum T and r sequences 
in some detail in Sec. VI] . 



Another way to view the range of T values is found in 
Fig. ^ in which we plot the distribution of T values for 
period-16 and period-23 sequences. Note the asymmetry; 
most T values are closer to the minimum T than the 
maximum T. Similar observations can be made about 
the distribution of r values shown in Fig. pi 
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FIG. 7. Normalized histogram of the distribution of T val- 
ues for period- 18 and period-23 sequences. 
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FIG. 8. Normalized histogram of the distribution of r val- 
ues for period-18 and period-23 sequences. 

Finally, note from Fig. |^ and Table ^ that the num- 
ber of distinct r values is not strictly monotonic with 
increasing periodicity. Curiously, the number of distinct 
r values decreases as the period goes from 11 to 12 and 
it remains constant in going from P — 5 to P — 6, while 
it increases for all other P increases. 



VII. MINIMUM, MAXIMUM, AND AVERAGE T 
AND T 

Here we develop analytical expressions for the extreme 
behaviors of the transient information and the synchro- 
nization time, and we determine how these extreme val- 
ues grow asymptotically with period. This analysis al- 
lows us to derive a number of important conclusions 
about which complementary structural features these 
quantities capture. 

First, some notational preliminaries. For a given pe- 
riod, we denote by (T) the average transient information. 



averaged over all distinct prime words. Similarly, Tmin 
and Tniax denote the minimum and maximum T values 
at a given period, respectively. We use similar notation 
for the average, maximum, and minimum r values. 



A. Maximum-T Sequence 

A sequence's transient information T is large if H{L) 
approaches its asymptotic value of E = log2 P slowly. 
This can be deduced graphically from Figs. |l| and |^. 
The H{L) curve that grows most slowly is that in which 
the distribution over subwords is maximally nonuniform, 
since the entropy of a random variable decreases as its 
distribution departs from uniformity. For a given period 
P, the sequence with the maximum value of T is one 
consisting of P— 1 O's followed by an isolated 1, since this 
sequence is the one whose distribution over subwords is 
as nonuniform as possible. This behavior is seen in one 
of the period-8 sequences shown in Fig. |^. 

A direct calculation for the maximal-T sequence yields: 



- max p / ^ 

n^2 



(21) 



To develop a functional form for T^ax we approximate 
the sum by an integral. After some work, one obtains. 



1 



[2-P-p2+41n(2) 



ln(16)P 

+ p2ln(P) + (l + P)2ln(l + P)] 



For large P, this scales as 



1 



-PlOgnP 



(22) 



(23) 



The values obtained from Eq. ( pi|) agree with the exact 
numerical results shown in Table In Fig. ^ we plot the 
approximation of Eq. ( |2^ ) and compare them with our 
exact numerical results. As expected, the approximation 
is quite good and improves at larger P. 

The synchronization time r for the maximum-T word 
can be calculated directly by noticing that one is synchro- 
nized as soon as one sees a 1, or after P— 1 measurements, 
whichever comes first. One obtains, for large P: 



P- 1 



(24) 



B. Minimum-T Sequences 

The minimum-T word is particularly easy to analyze 
when the period is a power of 2. We begin with this spe- 
cial case and use it to derive a simple form for Tmin as a 
function of the period. 
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The sequences with the minimum transient informa- 
tion Tmin correspond to those whose H{L) curves rise 
most quickly to E. This is achieved by a sequence whose 
distribution over subwords, at each subword length, is 
most nearly uniform and so gives high entropy: the mea- 
surements are most informative about the phase. 

When the period P = 2^ , where M is an integer, the 
distribution over subwords for the Tmin sequence is ex- 
actly uniform. Words with this property are known as 
de Bruijn sequences. A binary de Bruijn sequence of or- 
der fc is a binary circular string containing every binary 
substring of length k exactly once. For example, the lexi- 
cographically smallest order-3 binary de Bruijn sequence 
is 00010111. Note that this was one of the period-8 se- 
quences whose H{L) curve was plotted in Fig. |[ In that 
figure, the rapid convergence of H{L) to logg 8 = 3 is 
clearly seen. 
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FIG. 9. The minimum, maximum, and average transient 
information values plotted as a function of period. The solid 
points are our exact results; the dashed lines are the approx- 
imations developed in the text: Eq. ( p^ ) for Tmax, Eq. ( |2^ ) 
for Tniin, and Eq. (||) for (T). 

Because every subword appears with equal probability, 
H{L) = L, until L = log2 P. That is, H{L) converges lin- 
early to the excess entropy E = logj P. In other words, 
the source appears to be a fair coin until synchrony is 
achieved. At that time, exact predictability becomes pos- 
sible. This linear convergence can be seen in Fig. |. The 
transient information is thus given by the area between 
H{L) and log2 P; 



log^P 



(25) 



Evaluating the sum, one has 

1 
2 

Equivalently, since E = log2 P, we may write this as 



T,„i„ « ^ (log2 P + log2 P) 



(26) 



T,„i„ « i E (E + 1) 



(27) 



This result is exact for all P that are a power of 2 and 
serves as an excellent lower bound for P's that are not. 

For periods that are not a power of 2, the minimum-T 
words are those for which the distribution over subwords 
is as uniform as possible. For example, for P = 10 there 
are two prime words with minimum T: 0000101111 and 
0001011101. Each has Ti„i„ of 7.39483. Note that for 
each word, the distribution over subwords of length 1 is 
uniform, while the distribution over words of length 2 is 
as uniform as can be, given that there are 10 occurrences 
of subwords of length 2; for each, two length-2 subwords 
occur with a frequency of 3/10 and two occur with a 
frequency of 2/10. 

We have obtained an analytic expression for Trnin for 
sequences of general period P. The main idea, as stated 
above, is to distribute the frequencies of the subwords as 
uniformly as possible. How uniform this distribution is 
depends on how 2^ (the number of subwords of length-L) 
divides into the period P. Following this line of reason- 
ing, we obtain the following results. Let 



np{L) = 



P 
2^ 



(28) 



and 



7Vodd(i) = P mod 2^ , 

A^even(i) = 2^ - N,m{L) 



(29) 
(30) 



Here, \x\ denotes the floor oi x — the largest integer 
smaller than x. Similarly, [x] is the ceiling oi x — the 
smallest integer larger than x. 

The H{L) curve converges to E = log2 P by length L*, 
where 



L* = 



fog2-P' 



(31) 



Then, we find that the transient information is given by: 



where 



L* r 



E 

n=l 



(i* + l)l0g2P- 



(32) 



iVevc„(2.)^l0g2 



Values obtained using Eq. 
suits of Table || 



(33) 



|32|) agree with the exact re- 
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C. Average T 

In Fig. ^ we have also plotted (T), the average value 
of the transient information, as a function of the period 
P. These average values were also given in Table ||. The 
average values appear to be well approximated by 



(T) = ilog^P + logaP 



(34) 



This is an empirical fit. Note that (T) and Tnun of 
Eq. (|2^ ) are the same to leading order in P. This is 
not surprising, given the asymmetry in the distribution 
of T values evident in Fig. 0. 



D. Minimum-r Sequences 

For a given period P, we found that the sequences 
with the minimum transient information T,„in are the 
same as the sequences with the minimum synchroniza- 
tion time Tinin- As noted above, these sequences are those 
for which the distribution of subwords is most nearly uni- 
form at each subword length. Using this observation, it is 
possible to derive an analytic expression for Tmin among 
the sequences of period P. 

As discussed above when considering the minimum- 
T word, if P = 2^^, where M is an integer, then the 
minimum-T word is such that all subwords appear with 
equiprobability. In particular, each subword of length 
log2 P occurs exactly once. Thus, regardless of the phase 
in which one begins observing such a sequence, there is 
no ambiguity about the phase of the sequence after mak- 
ing log2 P observations. For observations shorter than 
log2 P, there will always be some uncertainty about the 
phase of the process. Thus, the synchronization time is: 



l0g2^ 



E 



(35) 



This result is exact when P = 2^^ and is an excellent 
approximation for other periods. 

As mentioned above, E is the total phase information 
stored in the periodic source; it is simply the Shannon en- 
tropy of the P possible phases. Recall that the entropy 
of a random variable sets a lower bound on the number of 
yes-no questions needed, on average, determine the value 
of that variable. The synchronization time t is equivalent 
to this average number of yes-no questions — measuring 
a binary symbol entails asking a yes-no question of the 
source. Eq. (|3^) thus shows that the periodic word with 
the minimum r saturates the lower bound set by the en- 
tropy. 

As was the case for Tmin, one can derive an expres- 
sion for Tmin for periods that are not a power of 2. As 
with the minimum-T word, in the minimum-r word the 
distribution over the subwords is as uniform as possible. 
In the case where P = 2^ + 1, P — 2 phases will syn- 
chronize in M observations. The remaining two phases. 



though, require an additional observation to synchronize; 
an additional symbol must be seen to distinguish between 
these two phases. This was the case, for example, for the 
period-5 case considered in Fig. ^. Here, P = 5 = 2^ -I- 1 
and M — 2. Three of the phases synchronize after two 
observations, while two phases synchronize after three 
observations. 

Generalizing this observation to P = 2^^ -t- N, we ob- 
tain the following result: 



= ^[2N{M + l) + {P-2N)M] , 



where 



and 



M = 



(36) 



(37) 



N = P mod 2^'^ . (38) 
Eq. (^) can also be written as O, seq. A061717]: 



1 

P 



log2 P' 



(39) 



The minimum-r values given by Eq. (p€ )agree with the 
exact numerical results shown in Table l|. For large P, 
Eq. (5^) scales as log2 P, in agreement with Eq. 



E. Maximum-T Sequence 

Recall that the maximum-T sequence consisted of P— 1 
O's followed by one 1. For example, the Tmax sequence is 
000000001 for P = 9. However, this is 77.oi the maximum- 
r sequence. The sequence 000000001 has an H{L) that 
grows very slowly, leading to its large T. However, once 
one sees a 1, one is synchronized. Thus, it is possible 
for an observer to get lucky, see a 1 after one or two 
observations and be synchronized quickly. As a result, 
the expected synchronization time for this sequence is 
relatively low, only r = 4.88889. 

In contrast, the sequence with the largest synchroniza- 
tion time for period 9 is 000010001 and has r = 6.2222. 
This sequence has the maximum r due the presence of 
an additional 1 which prevents an observer from synchro- 
nizing after a single observation; this leads to a larger r. 

The maximum-r word for periods 4-16 are given in 
Table [II. Note that a maximum-r word need not be 



unique, although wc find that it is unique for all periods 
examined, except for P = 6. 

For P odd, we noticed the maximum-r word takes on a 
particularly simple form: two I's separated by (P — 3)/2 
zeros. This allows us to determine a simple expression 



W = |P - ^ - ^ , P Odd 



(40) 
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If the period is even, it is also possible to write down 
an expression for Tmax- If P is divisible by 4, then 



-P 



P divisible by 4 



1 1 

4' 2 " P 

However, if P is divisible by 2, but not by 4, then: 



(41) 



-P 



1 
2 



4 



P divisible by 2 and not 4 . (42) 



Note that, in all cases, to leading order in P, t„ 
linearly with P: 



grows 



(43) 



Also, in all cases the preceding results agree with the ex- 
act values obtained by enumeration and shown in Tables 
U and m. Eqs. ^ through (H) were obtained by not- 
ing various regularities in the enumeration data and not 
via a first-principles calculation of the subword statistics. 

It is also possible to calculate the transient information 
T for the word that maximizes r for the case in which P 
is odd. Recall that the Tmax word in this case consisted 
of two I's separated by (P — 3)/2 zeros. A brute force, 
direct counting approach yields, for large P: 



Tmax ^ ^ ^ PE -|- ^ 



(44) 
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FIG. 10. The minimum, maximum, and average synchro- 
nization time plotted as a function of period. The solid points 
are the exact results from the enumeration; the dashed lines 
are the approximations developed in the text. The approx- 
imation for Tjnax is that of Eq. (|^, which is exact for odd 
periods. The approximation for Tmin is that of Eq. (p^, which 
is exact for periods that are a power of 2. The approximation 
for (T) is the empirical fit of Eq. (jisl). 



F. Average Synchronization Time (r) 

Figure ^ also shows (r) , the average synchronization 
time at a given period, as a function of the period. The 
dashed line in this figure is an empirical fit: 



(r) « -logsP 



(45) 



As was the case with the transient information, the aver- 
age and minimum synchronization times appear to grow 
asymptotically at the same rate. 



VIII. RELATIONSHIPS BETWEEN r AND T 

Throughout the foregoing we have argued that there 
are important structural differences between distinct pe- 
riodic sequences at a given period. Moreover, the struc- 
tural properties captured T and r, though both related to 
synchronization, can be different. Perhaps the differences 
are not surprising. Based on their definitions these quan- 
tities have different interpretations: T measures the total 
uncertainty experienced while synchronizing; t measures 
the expected number of observations needed in order to 
synchronize. To clarify the differences and similarities, 
in this section we analyze more directly the relationship 
between T and r. We shall see that, while there are 
relationships between T and r for the extremal synchro- 
nization behaviors, there is a fairly wide range of T and r 
combinations possible among different words of the same 
period. 



Period 




Tmax word 


4 


2.25 


0001 


5 


3.2 


00101 


6 


3.3333 


000101, 000001 


7 


4.71429 


0001001 


8 


5.375 


00100101 


9 


6.222 


000010001 


10 


6.6 


0001001001 


11 


7.72727 


00000100001 


12 


8.41667 


001010010101 


13 


9.23077 


0000001000001 


14 


9.71429 


00001000010001 


15 


10.7333 


000000010000001 


16 


11.4375 


0010101001010101 


TABLE III. 


Maximum synchronization-time words and 


Tmax values as 


a function of period P. 




Function 


Minimum 


Maximum 


T 


ilog^P=iE2 


£log2P=£E 


r 


log2 P = E 





TABLE IV. Summary of minimum and maximum T and r 
behaviors expressed as a function of period P and as a func- 
tion of the excess entropy E. 
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A. Extreme Synchronizations 



Er, 



max T ■ 



(47) 



We start by summarizing the behaviors of the mini- 
mum and maximum values of the transient information 
T and the synchronization time r to leading order in the 
period P in Table IV. Note the enormous range in T 
and T values. For example, if P = 256, Tmin ~ 32, while 
Tmax ~ 1024. Similarly, for P = 256, r^in ~ 8 while 

'^max ~ 192. 

Note that for both the minimum and maximum cases, 
T and r are related to each other by: 



T = otE , 



(46) 



where a is a constant that does not depend on the pe- 
riod P. For the minimum- (T, r) case a = 1/2 and for 
the maximum case a = 2/3. Recall, however, that the 
maximum-T word is not the same as the maximum-r 
word. 



If one uses Tmax and Tmax T from Eq. 

1. And, if one uses Tm; 



j m 
and 



Eq. (46), one finds a = 
Tmax, one find a = 1/3. 

In all cases, note that the ratio of T to E gives a quan- 
tity proportional to r. This ratio may thus be viewed as 
setting a characteristic time for synchronization. 



To maximize T for a given r, assume that the given t is 
that which maximizes T. Then, using the above equa- 
tion, we see that Tuppor oc Er. That is, the slope of the 
linear upper bound is E. To get the full equation of the 
line, we require that it go through the minimum- (r, T) 
point on the lower left hand corner of Fig. |l^. Using 
Eqs. ( p6[ ) and (35) for the values of T„ijn and Tmi,,, re- 
spectively, we t 



len obtain 



T„pp„ « E[r--(E-1) 



(48) 



This upper bound is plotted in Fig. ^ with E 
log2l8 « 4.1699 . . .. This upper bound is only approxi- 
mate because the minimum- (r, T) values used in its cal- 
culation are the asymptotic values for large P. The 
bound is also approximate because the relationship in 
Eq. (|4^) , which is true for the maximum-T sequence (the 
solid triangle in the figure), is assumed to hold for all se- 
quences. Despite these approximations, the linear upper 
bound appears to fit the data quite well, as shown in 
Fig. 0. 



B. Nonextreme Synchronizations 

It turns out, however, that these simple relations mask 
a wide diversity of synchronization behaviors. Fig. |ll|, a 
scatter plot of T and t for all period-18 sequences, shows 
that the relationship between T and r for individual se- 
quences is quite a bit more complicated. For example, 
recall from the previous section that the maximum-T 
sequence does not correspond with the maximum-r se- 
quence. In the Fig. |l|, the maximum-T value is shown 
by a solid triangle and the maximum-r value by a solid 
square. The minimum-T and -r sequences are identi- 
cal, however. This corresponds to the single point in the 
figure's lower left corner. 

The wide range of points in the scatter plot of Fig. [ll| 
makes it clear that T and r do indeed measure diff'er- 
ent properties: T and r are not simply rescaled versions 
of each other. Any simple functional relationship is pre- 
cluded by the diffuse scatter of points. Said another way, 
if one lists sequences of a given period in order of in- 
creasing T, this order will not be the same as listing the 
sequences in order of increasing r. 

Although there is considerable spread evident in 
Fig. 11, there are bounds limiting the range of T val- 
ues at each r. 

In fact, one can develop an approximate form for the 
upper bound in Fig. ^ as follows. Given an arbitrary r 
value and the period P, we are interested in determining 
the largest possible T. Call this maximum value Tapper- 
Recall that the synchronization time r^ax t for the max- 
imum T word scales as P/2 for large P, as shown in 
Eq. (p3). This lets us write 
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FIG. 11. A scatter plot of T versus r for all period-18 se- 
quences. The upper bound is that of Eq. (j^j and the lower 
bound is that of Eq. (|5o|). The solid triangle is the (r, T) 
point for the Tmax sequence; the solid square denotes the 
Tmax sequence. 

A similar line of reasoning allows us to form a lower 
bound Tiower for the smallest possible T, given a value 
of r and the period P. From Eqs. (p7|) and (pq), for large 



P we know that T„ 
us 



lE'^ and t„ 



-Er^ 



E. This gives 



(49) 



To minimize T, we assume that the given r is the min- 
imum r. This tells us that the slope of a linear lower 
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bound is ^E. Requiring the line to go through the mini- 
mum T, T point yields 

Tlowcr = \^+ ■ (50) 

This is only an approximate bound, for the same rea- 
sons that Tuppor is approximate. Eq. ( [5C| ) is plotted in 
Fig. |ll|. It is indeed a lower bound, but the bound is not 
very tight. 

IX. DISCUSSION AND CONCLUSION 

We began by introducing two measures of how difhcult 
it is to synchronize to a sequence: the transient infor- 
mation T, an information-theoretic measure of the total 
uncertainty experienced by an observer during the syn- 
chronization process; and the synchronization time t, the 
number of symbols an observer expects to measure be- 
fore it is synchronized. We exactly calculated T and r 
for all synchronization-distinct periodic sequences up to 
and including period 23. We also derived a handful of an- 
alytic expressions and approximations for the minimum, 
maximum, and average T and r as a function of period. 

Our results show that there are many structural differ- 
ences between periodic sequences, even within sequences 
of the same period. These differences are simply not 
captured by commonly used information theoretic mea- 
sures, such as the entropy rate and the excess entropy. 
Are all periodic sequences of a given period the same 
from an information-theoretic standpoint? The answer 
we gave here is "no"; the recently introduced transient 
information, in fact, captures the information-theoretic 
differences between sequences of a given period. Are 
there structural differences between different periodic se- 
quences of the same period? We showed that the answer 
is "yes" and argued that the differences are significant. 
Are there distinct classes of periodic sequence? Again, we 
provided a positive answer, partly by contrasting prop- 
erties captured by the transient information with a se- 
quence's group theoretic properties and its synchroniza- 
tion time. The latter measures the average number of 
observations needed to synchronize to the sequence and, 
for a given sequence at a given period, this number was 
neither the same nor simply proportional to the transient 
information. Nonetheless, at a coarse level, the leading- 
order approximations we developed provide simple and 
direct links between the three different concepts of phase 
memory E, synchronization time r, and the total uncer- 
tainty experienced during synchronization T. 

There are a number of areas in which our results may 
find application. For one, recognizing the phases of long 
periodic sequences of the type investigated here is a key 
technology for current and future large-scale satellite- 
to-satellite communication systems. These multiagent 
systems require extremely accurate and robust satellite- 
to-satellite synchronization generally for coordina- 
tion in signal processing and navigation and specifically 



for estimating inter-satellite signal delays and distances. 
Synchronization between two satellites is effected by one 
transmitting a very long {P ^ 1,000) and known bi- 
nary periodic sequence. The receiver then infers the 
phase, using a hierarchical correlation algorithm adapted 
to account for noise corruption of individual symbols 
during transmission. It turns out that the long peri- 
odic acquisition sequences are aperiodic sequences formed 
from relatively "prime" short aperiodic sequences. In 
fact, the synchronization properties of the component 
sequences and the composite acquisition sequence deter- 
mine bounds on the computational effort and noise ro- 
bustness that can be achieved by the receiver's detection 
system. We conjecture that the transient information of 
the acquisition sequence is a key parameter determining 
these properties. 

More generally, these results bear on any situation in 
which an intelligent agent needs to learn the phase of 
a periodic component in its environment's behavior; cf. 
Ref. This arises in multiagent systems if an agent 
needs to know where it is in a physical environment that 
varies periodically in space or time, if it needs to ad- 
just its behavior in response to the repetitive behavior of 
other agents, or if a collective decision requires coordi- 
nated information processing. 

Finally, we mention several open questions and di- 
rections for future work. First, do these results hold 
if the synchronization process is noisy and bits flip oc- 
casionally? They should extend relatively directly to a 
class of "noisy-periodic" processes in which one or sev- 
eral symbols in a periodic sequence are random. How- 
ever, can these results be extended to the full class of 
finite-memory sources? For a further discussion of the 
synchronization scenario in a settiiig not restricted to pe- 
riodic sequences see Refs. § and g. 

Second, one may view those sequences at a given pe- 
riod with the same T as forming an equivalence class. Re- 
call that distinct periodic sequences were defined by the 
group S'fc X Cp. These are the equivalence classes of zero 
entropy rate, finite excess entropy sequences. But what 
new algebra or symmetry group of periodic sequences 
characterizes those sequences that are equally hard to 
synchronize to, as measured by T or r? If we can charac- 
terize identical T or t sequences in terms of a symmetry 
group or algebra, it should be possible to obtain an ana- 
lytic expression for the number of distinct T or r values 
at a given period. 

Third, as noted above, Titsworth pointed out that 
Eq. ( [isl ) gives the number of distinct finite-state machines 
needed to generate all sequences of a given prime period. 
The picture is that starting in a different state of such 
a machine gives the cyclic permutations, while exchang- 
ing the output symbols gives the permutation <-> 1. 
Titsworth does not comment on the structure of the syn- 
chronizing states of these machines. To account for syn- 
chronization one wants each finite-state machine to have 
a unique start state that corresponds to the condition of 
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an observer not knowing in which phase a per iodic se- 
quence starts. It turns out that e-machines [po|-p2[ cap- 
ture this synchronizing structure in their transient states, 
and these in turn determine T. Moreover, this is true 
for both periodic and random sources. Based on these 
observations we conjecture that the number of distinct 
T values for a given period is the number of distinct e- 
machines needed to recognize all periodic sequences of 
a given prime period. That is, we conjecture that those 
sequences with the same T value have the same transient 
e-machine structure, up to edge relabeling. 

In summary, we analyzed what is arguably the sim- 
plest class of sequences: zero entropy rate, finite-period 
sequences. The sequences represent behaviors that are 
often ignored in many fields as being too simple. As we 
demonstrated, however, their structural complexity and 
synchronization properties are rich and subtle. Within 
a given period, there are large differences in structural 
complexity, as measured by the transient information T 
and the synchronization time t. Thus, even simple peri- 
odic systems are surprisingly complex. 
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